feat: add --inject flag to codeflash compare#1985
Conversation
When benchmarking already-merged optimizations, the benchmark file often doesn't exist at either the base or head ref. The --inject flag copies specified files/directories from the working tree into both worktrees before benchmark discovery and execution, eliminating the need to cherry-pick benchmark commits onto temporary branches. Usage: codeflash compare <base> <head> --inject tests/benchmarks/test_bench.py
|
Claude finished @KRRT7's task in 1m 58s —— View job PR Review Summary
Prek Checks
Mypy errors in Code ReviewThe implementation is clean and correct. Two minor findings: 1. Inline The file already has several inline 2.
No bugs, security issues, or breaking API changes found. Duplicate DetectionNo duplicates detected. The inject logic is new and localized to Bot PRsPR #1954 ( Last updated: 2026-04-03T16:24Z |
- Replace `import shutil as _shutil` with plain `import shutil` to match the existing style in the same function - Warn when --inject is used with --script mode (unsupported combo) instead of silently dropping the flag
1 participant
--injectflag tocodeflash comparethat copies files/directories from the working tree into both worktrees before benchmark discovery and executionProblem
When running
codeflash compare <base_ref> <head_ref>, codeflash creates git worktrees at those refs. If the benchmark test was written after both refs — the typical case when benchmarking a merged optimization — the benchmark doesn't exist in either worktree and the comparison fails with "no tests ran."The existing
benchmarks_rootcopy logic insiderun_benchmark_on_worktree()partially addresses this, but only for the benchmarks directory as a whole and only in the memory-only path. The--injectflag provides explicit, general-purpose file injection that works for any file (benchmark tests, pyproject.toml config, conftest.py, etc.).Usage
Paths are relative to the repo root. Files are copied into the same relative path in both worktrees. Missing source paths emit a warning and are skipped.
Test plan
codeflash compare 189a6124 62dd690b --memory --inject tests/benchmarks/test_benchmark_table_extraction_memory.py pyproject.tomlsucceeds where it previously failed with "no tests ran"